这是indexloc提供的服务,不要输入任何密码
Skip to content

add native queries, functions or virtual collections defined by pipelines #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 38 commits into from
Apr 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
c3fa8bb
configuration for native queries
hallettj Apr 5, 2024
d000d2f
update tests to accept database in query executor instead of collection
hallettj Apr 6, 2024
4c6269b
begin query plan with top-level native query
hallettj Apr 10, 2024
4e86df9
update fixtures with a basic native query
hallettj Apr 10, 2024
2a14c9b
give query request conversion access to function result types
hallettj Apr 10, 2024
fab2329
update query conversion tests
hallettj Apr 10, 2024
80877fd
wip:
hallettj Apr 11, 2024
29dd28f
Merge branch 'main' into jesse/native-queries-using-pipelines
hallettj Apr 15, 2024
03207c4
rename test helper
hallettj Apr 15, 2024
8ec01c3
log query request target
hallettj Apr 15, 2024
63e42c3
interpolate native query arguments
hallettj Apr 15, 2024
190feed
make QueryConfig implement Copy
hallettj Apr 15, 2024
79c1efa
update fixture
hallettj Apr 15, 2024
153ec52
unit test for native query
hallettj Apr 16, 2024
1e9c769
move native query fixture to default connector config
hallettj Apr 16, 2024
4c12875
avoid crate rebuilds when fixtures change
hallettj Apr 16, 2024
6aa94b6
support representations as collections or as functions
hallettj Apr 16, 2024
61a48eb
include native queries in schema response
hallettj Apr 16, 2024
5705911
push configuration processing into configuration crate
hallettj Apr 18, 2024
006e89c
pass around Configuration instead of MongoConfig
hallettj Apr 18, 2024
9979b38
wip: pass around Configuration instead of MongoConfig
hallettj Apr 18, 2024
9203c55
wip: remove dead schema code from common, config type changes
hallettj Apr 18, 2024
efea08f
wip: QueryContext copies from Configuration
hallettj Apr 18, 2024
df6a388
wip: match ndc lib revs
hallettj Apr 18, 2024
532c899
Merge branch 'main' into jesse/native-queries-using-pipelines
hallettj Apr 18, 2024
6878b0d
match ndc-models tag from sdk
hallettj Apr 18, 2024
9dd80fb
wip: configuration types
hallettj Apr 18, 2024
afd0a97
configuration changes compile - next up, tests
hallettj Apr 18, 2024
e2c4cfa
update tests for type changes
hallettj Apr 18, 2024
8f30bfd
lint fixes
hallettj Apr 18, 2024
c088dcb
fix native query fixture
hallettj Apr 18, 2024
91af19d
we need to preserve function object types for query request processing
hallettj Apr 19, 2024
6e4bcf9
remove debugging output
hallettj Apr 19, 2024
be8cbcd
add missing list item dash in error message
hallettj Apr 19, 2024
db180e8
change field name from "type" to "result_document_type"
hallettj Apr 19, 2024
9db1014
oh wait - for consistency that should be camelCase
hallettj Apr 19, 2024
081adac
remove commented-out code
hallettj Apr 19, 2024
cc1f467
combine FunctionInfos and CollectionInfos into one map
hallettj Apr 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 10 additions & 33 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ members = [
]
resolver = "2"

# The tag or rev of ndc-models must match the locked tag or rev of the
# ndc-models dependency of ndc-sdk
[workspace.dependencies]
ndc-sdk = { git = "https://github.com/hasura/ndc-sdk-rs.git" }
ndc-models = { git = "http://github.com/hasura/ndc-spec.git", tag = "v0.1.2" }

# We have a fork of the mongodb driver with a fix for reading metadata from time
# series collections.
# See the upstream PR: https://github.com/mongodb/mongo-rust-driver/pull/1003
Expand Down
12 changes: 6 additions & 6 deletions crates/cli/src/introspection/sampling.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ use configuration::{
};
use futures_util::TryStreamExt;
use mongodb::bson::{doc, Bson, Document};
use mongodb_agent_common::interface_types::MongoConfig;
use mongodb_agent_common::state::ConnectorState;
use mongodb_support::BsonScalarType::{self, *};

type ObjectField = WithName<schema::ObjectField>;
Expand All @@ -19,18 +19,18 @@ type ObjectType = WithName<schema::ObjectType>;
/// are not unifiable.
pub async fn sample_schema_from_db(
sample_size: u32,
config: &MongoConfig,
state: &ConnectorState,
existing_schemas: &HashSet<std::string::String>,
) -> anyhow::Result<BTreeMap<std::string::String, Schema>> {
let mut schemas = BTreeMap::new();
let db = config.client.database(&config.database);
let db = state.database();
let mut collections_cursor = db.list_collections(None, None).await?;

while let Some(collection_spec) = collections_cursor.try_next().await? {
let collection_name = collection_spec.name;
if !existing_schemas.contains(&collection_name) {
let collection_schema =
sample_schema_from_collection(&collection_name, sample_size, config).await?;
sample_schema_from_collection(&collection_name, sample_size, state).await?;
schemas.insert(collection_name, collection_schema);
}
}
Expand All @@ -40,9 +40,9 @@ pub async fn sample_schema_from_db(
async fn sample_schema_from_collection(
collection_name: &str,
sample_size: u32,
config: &MongoConfig,
state: &ConnectorState,
) -> anyhow::Result<Schema> {
let db = config.client.database(&config.database);
let db = state.database();
let options = None;
let mut cursor = db
.collection::<Document>(collection_name)
Expand Down
11 changes: 7 additions & 4 deletions crates/cli/src/introspection/validation_schema.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,22 @@ use configuration::{
};
use futures_util::TryStreamExt;
use mongodb::bson::from_bson;
use mongodb_agent_common::schema::{get_property_description, Property, ValidatorSchema};
use mongodb_agent_common::{
schema::{get_property_description, Property, ValidatorSchema},
state::ConnectorState,
};
use mongodb_support::BsonScalarType;

use mongodb_agent_common::interface_types::{MongoAgentError, MongoConfig};
use mongodb_agent_common::interface_types::MongoAgentError;

type Collection = WithName<schema::Collection>;
type ObjectType = WithName<schema::ObjectType>;
type ObjectField = WithName<schema::ObjectField>;

pub async fn get_metadata_from_validation_schema(
config: &MongoConfig,
state: &ConnectorState,
) -> Result<BTreeMap<String, Schema>, MongoAgentError> {
let db = config.client.database(&config.database);
let db = state.database();
let mut collections_cursor = db.list_collections(None, None).await?;

let mut schemas: Vec<WithName<Schema>> = vec![];
Expand Down
9 changes: 4 additions & 5 deletions crates/cli/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,9 @@ use std::path::PathBuf;

use clap::{Parser, Subcommand};

use mongodb_agent_common::interface_types::MongoConfig;

// Exported for use in tests
pub use introspection::type_from_bson;
use mongodb_agent_common::state::ConnectorState;

#[derive(Debug, Clone, Parser)]
pub struct UpdateArgs {
Expand All @@ -29,7 +28,7 @@ pub enum Command {

pub struct Context {
pub path: PathBuf,
pub mongo_config: MongoConfig,
pub connector_state: ConnectorState,
}

/// Run a command in a given directory.
Expand All @@ -44,14 +43,14 @@ pub async fn run(command: Command, context: &Context) -> anyhow::Result<()> {
async fn update(context: &Context, args: &UpdateArgs) -> anyhow::Result<()> {
if !args.no_validator_schema {
let schemas_from_json_validation =
introspection::get_metadata_from_validation_schema(&context.mongo_config).await?;
introspection::get_metadata_from_validation_schema(&context.connector_state).await?;
configuration::write_schema_directory(&context.path, schemas_from_json_validation).await?;
}

let existing_schemas = configuration::list_existing_schemas(&context.path).await?;
let schemas_from_sampling = introspection::sample_schema_from_db(
args.sample_size,
&context.mongo_config,
&context.connector_state,
&existing_schemas,
)
.await?;
Expand Down
7 changes: 5 additions & 2 deletions crates/cli/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -45,10 +45,13 @@ pub async fn main() -> anyhow::Result<()> {
Some(path) => path,
None => env::current_dir()?,
};
let mongo_config = try_init_state_from_uri(&args.connection_uri, &Default::default())
let connector_state = try_init_state_from_uri(&args.connection_uri)
.await
.map_err(|e| anyhow!("Error initializing MongoDB state {}", e))?;
let context = Context { path, mongo_config };
let context = Context {
path,
connector_state,
};
run(args.subcommand, &context).await?;
Ok(())
}
1 change: 1 addition & 0 deletions crates/configuration/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ futures = "^0.3"
itertools = "^0.12"
mongodb = "2.8"
mongodb-support = { path = "../mongodb-support" }
ndc-models = { workspace = true }
schemars = "^0.8.12"
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1" }
Expand Down
Loading