ProSuite package

Data Model

class prosuite.data_model.BaseDataset(name: str, filter_expression: str = '')

Bases: object

The base class for datasets representing tabular data. It is either a Dataset or, in the future, a TransformedDataset

name: str

The name of the dataset. Must be unique with respect to all other datasets of the model.

class prosuite.data_model.Dataset(name: str, model: Model, filter_expression: str = '')

Bases: BaseDataset

A dataset represents data from a table or feature class in a workspace, optionally filtered by an SQL expression.

Parameters:

  • name (str) – table or featureclass name

  • model (class:prosuite.data_model.Model) – The prosuite.data_model.Model this dataset belongs to.

  • filter_expression (str, optional) – A where clause that filters the table. The syntax of the where clause is defined in the document SQLSyntax_en.pdf

property alias_name: str

The alias name of the dataset in the data dictionary, or None it has not been loaded from the data dictionary. This property is read-only.

property id: int

The unique identifier of the dataset in the data dictionary, or None it has not been loaded from the data dictionary. This property is read-only.

model: Model

The dataset’s data model.

class prosuite.data_model.Model(name, catalog_path)

Bases: object

The Model represents the data model in a workspace (file-gdb or enterprise-gdb)

Parameters:

  • name (str) – name of the model

  • catalog_path – The catalog path of the associated workspace.

catalog_path examples:

c:/data.gdb c:/enterprise_gdb.sde

add_dataset(dataset)

Add a dataset to the model.

catalog_path: str

The catalog path of the associated workspace.

datasets

List of datasets in the model.

property default_database_name: str

Gets the default database name for this model as defined in the data dictionary, or None it has not been loaded from the data dictionary. This property is read-only.

property default_database_schema_owner: str

Gets the default database schema owner for this model as defined in the data dictionary, or None it has not been loaded from the data dictionary. This property is read-only.

property element_names_are_qualified: bool

Gets whether Whether model element names (datasets, associations) are qualified in this model. For models not loaded from the data dictionary always fully qualify dataset/table names from RDBMS. This property is read-only.

get_dataset_by_name(name)

Get a dataset from this model by its name.

property id: str

The unique identifier of the model in the data dictionary, or None it has not been loaded from the data dictionary. This property is read-only.

name: str

The unique name of the model.

property wkid: int

The spatial reference well-known ID (WKID) for the model as defined in the data dictionary, or None it has not been loaded from the data dictionary. This property is read-only.

class prosuite.data_model.TransformedDataset(transformer_descriptor: str, name: str = '', filter_expression: str = '')

Bases: BaseDataset

A transformed dataset represents tabular data that has been transformed from other input tables or feature classes, optionally filtered by an SQL expression.

Parameters:

  • transformer_descriptor (str) – The transformer descriptor, i.e. the algorithm used to generate this dataset.

  • name (str) – name of the transformed table or featureclass

  • filter_expression (str, optional) – A where clause that filters the table. The syntax of the where clause is defined in the document SQLSyntax_en.pdf

generate_name()

Generates a technical name using the dataset name(s) and the transformer descriptor. This is the default name of a transformed dataset if it was created by the standard factory method from prosuite.factories.transformers.

generate_name_with_timestamp()

Generates a technical name using the dataset name(s) and the transformer descriptor, and appends a timestamp to ensure uniqueness.

name: str

The unique name of the transformed dataset.

parameters: List[Parameter]

The list of parameters. Typically the parameters are specified in the factory method used to create the transformed dataset (see prosuite.factories.transformers.Transformers) and do not need to be changed through this list.

transformer_descriptor

The transformer descriptor, i.e. the algorithm used to generate this dataset.

Quality

class prosuite.quality.Condition(test_descriptor: str, name: str = '')

Bases: object

A quality condition. Conditions are the primary building blocks of quality specifications. Each Condition is a configuration of a test algorithm for one or more datasets. Conditions must be created with the factory methods from prosuite.factories.quality_conditions

category: str

The name of the category, if this issue is assigned to a category.

description: str

Freely definable description of the Quality Condition. This description can be displayed when viewing issues in the issue navigator, and may contain explanations to the Quality Condition and instructions for correcting the issues.

generate_name()

Generates a technical name using the dataset name(s) and the test descriptor. This is the default name of a condition if it was created by the standard factory method from prosuite.factories.quality_conditions.

property id: int

The unique identifier of the condition in the data dictionary, or None if it has not been loaded from the data dictionary. This property is read-only.

issue_filter_expression: str

The filter expression to be used for multiple issue_filters.

issue_filters: List[IssueFilter]

The list of issue filters for this condition.

issue_type: IssueType

Defines if a failing test returns a warning or an error issue. Quality conditions with Issue Type = Warning are considered “soft” conditions, for which exceptions (“Allowed Issues”) may be defined.

name: str

The unique name of the quality condition.

parameters: List[Parameter]

The list of parameters. Typically the parameters are specified in the factory method used to create the quality condition (see prosuite.factories.quality_conditions) and do not need to be changed through this list.

stop_on_error: bool

Indicates if the occurrence of an error for an object should stop any further testing of the same object. This can be used to prevent further tests on a feature after a serious geometry error (e.g. incorrectly oriented rings) was detected for the feature. The used Test Descriptor provides a standard value for this property. It can optionally be overridden here.

test_descriptor: str

The test descriptor, i.e. the algorithm used to verify this condition.

url: str

Optional URL to a website providing additional information for this Quality Condition. Certain Quality Conditions require more detailed information about the test logic and/or the correction guidelines than the field “Description” can provide. This information can for example be assembled in a wiki, and the URL may be provided here. When viewing issues in the issue navigator, the corresponding web page can be opened. In the HTML verification reports these URLs are used to render the names of the Quality Conditions as links.

class prosuite.quality.DdxSpecification(ddx_id: int, project_short_name: str)

Bases: object

Represents a specification defined in the Data Dictionary (DDX). This requires a server that is configured with a data dictionary connection.

ddx_id: int

ID of the specification in the data dictionary. Find out by using the properties window of the specification in the data dictionary editor.

project_short_name: str

Short name of the project in the data dictionary. This is used to determine the main main workspace of the verification. In case the issues are written to the error datasets in the data model the project short name is used to determine the correct error datasets. See update_issues_in_verified_model

class prosuite.quality.IssueFilter(issue_filter_descriptor: str, name: str = '')

Bases: object

An issue filter optionally filters issues found by a quality condition by configurable criteria. Its purpose is the reduction of false positives. Each quality condition can have none, one or multiple issue filters.

Parameters:

  • issue_filter_descriptor (str) – The issue filter descriptor, i.e. the algorithm used to filter issues found by the condition.

  • name (str) – name of the issue filter

description: str

The description of the issue filter.

generate_name()

Generates a technical name using the dataset name(s) and the test descriptor. This is the default name of a condition if it was created by the standard factory method from prosuite.factories.quality_conditions.

property id: int

The unique identifier of the issue filter in the data dictionary, or None if it has not been loaded from the data dictionary. This property is read-only.

issue_filter_descriptor: str

The issue filter descriptor, i.e. the algorithm used to filter the issues.

name: str

The (unique) name of the issue filter.

parameters: List[Parameter]

The list of parameters. Typically the parameters are specified in the factory method used to create the issue filter (see prosuite.factories.issue_filters) and do not need to be changed through this list.

class prosuite.quality.IssueType(value)

Bases: Enum

Error = 1
Warning = 0
class prosuite.quality.Parameter(name: str, value)

Bases: object

A parameter configures a quality condition. Parameters can represent Datasets (dataset parameter) or scalar values (scalar parameters). Parameters have a name and a value.

Dataset parameters: value is of type dataset. The parameter can retrieve the workspace id (model name or model id if it has been retrieved from the data dictionary) and the where clause (filter expression) of the dataset.

Scalar parameters: value is a simple type (number, string, bool).

property dataset

The dataset associated with this parameter (if any).

get_string_value() → str
get_where_clause()
get_workspace_id()
is_dataset_parameter() → bool

Check if the parameter is a dataset parameter. A dataset parameter is a parameter that has a Dataset, a TransformedDataset or a list of datasets as its value. :return: True if the parameter is a dataset parameter, False otherwise.

is_dataset_value() → bool

Check if the parameter value is a Dataset (as opposed to a TransformedDataset or a list). :return: True if the parameter is a dataset parameter, False otherwise.

is_transformed_dataset_value() → bool

Check if the parameter value is a TransformedDataset (as opposed to a Dataset or a list).”

name: str

The parameter name.

property value

The parameter value. Can be a dataset, transformed dataset, list of datasets, or a scalar value.

static value_is_list_of_datasets(value)
class prosuite.quality.Specification(name: str = 'Custom Specification', description: str = '')

Bases: object

Represents a quality specification defined in code. Quality specifications are collections of quality conditions.

Parameters:

  • name (str) – specification name

  • description (str) – specification description

add_condition(condition: Condition)

Adds conditions to the specification

get_conditions() → List[Condition]

Returns the List of conditions

project_short_name: str

Short name of the project in the data dictionary. If the project short name is provided, the datasets registered in the data dictionary are used. Otherwise, the datasets are retrieved using the provided catalog path of the datasets’ model. The project short name is also used to determine the main workspace of the verification. The project short name is also used to determine the correct error datasets in the data model in case the issues are stored in the error datasets of the data model. See update_issues_in_verified_model In case the issues are written to the error datasets in the data model the project short name is used to determine the correct error datasets. See update_issues_in_verified_model

class prosuite.quality.XmlSpecification(specification_file: str, specification_name: str = None, data_source_replacements: List[List[str]] = None)

Bases: object

Represents a specification defined in the xml specification schema. Quality specifications are collections of quality conditions.

Parameters:

  • specification_file (str) – path to the xml specification file

  • specification_name (str) – name of the specification (in the xml file) that should be executed. If not defined in the constructor, it needs to be defined before it is used.

  • data_source_replacements ([[str]]) – a list containing a list with two string elements. These represent a workspace id together with the path to the workspace. Example: [[“TLM_Production”, “C:/temp/user@topgist.sde”]]. More details are provided below the XML verification sample.

static get_specification_names(specification_file: str) → List[str]

Returns a list of the specification names of all specifications in all categories of the xml-specifications doc.

Parameters:

specification_file (str) – path of xml specification file.

Returns:

List of specification names

Return type:

[str]

Verification

class prosuite.verification.EnvelopePerimeter(x_min: float, y_min: float, x_max: float, y_max: float)

Bases: object

A spatial envelope defined by the bounding coordinates. The spatial reference must match the spatial reference of the datasets.

class prosuite.verification.EsriShapePerimeter(esri_shape: bytes)

Bases: object

A polygon in the Esri shape buffer format.

class prosuite.verification.InvolvedTable(table_name: str, object_ids: list)

Bases: object

Represents a table involved in a verification issue.

object_ids

A list of object IDs from the table that are involved in the issue.

table_name

The name of the table.

class prosuite.verification.Issue(issue_msg: prosuite.generated.shared_qa_pb2.IssueMsg)

Bases: object

Represents an issue found during the verification.

To initialize an Issue object, pass the issue message to it. The Issue object extracts some properties from the issue message and makes them available as attributes:

allowable

If the issue is allowable.

description

The description of the issue.

geometry

The geometry involved of the issue.

involved_objects

A list of InvolvedTable objects that are involved in the issue.

issue_code

The issue code ID.

stop_condition

If the issue is a stop condition.

class prosuite.verification.MessageLevel

Bases: object

level_10000 = 'Verbose'
level_110000 = 'Fatal'
level_30000 = 'Debug'
level_40000 = 'Info'
level_60000 = 'Warn'
level_70000 = 'Error'
class prosuite.verification.Service(host_name: str, port_nr: int, channel_credentials: grpc.ssl_channel_credentials = None)

Bases: object

The service class communicates on the http/2 channel with the server and initiates the quality verifications.

HTML_REPORT = 'verification.html'

The name of the html verification report. It will be written to the output_dir specified in the prosuite.service.Service.verify() method.

ISSUE_GDB = 'Issues.gdb'

The name of the issue File Geodatabase. It will be written to the output_dir specified in the prosuite.service.Service.verify() method. This File Geodatabase contains the issues found during the verification and could be used as the source for the Issue Worklist in the ProSuite QA Add-In for ArcGIS Pro.

XML_REPORT = 'verification.xml'

The name of the xml verification report. It will be written to the output_dir specified in the prosuite.service.Service.verify() method.

configure_parameter_msgs(parameters, parameter_msgs)
host_name

The name or IP address of the host running the quality verification service.

port_nr

The port used by the quality verification service.

ssl_channel_credentials: grpc.ssl_channel_credentials

The channel credentials to be used for TLS/SSL server authentication, if required by the server (Default: None -> No TLS/SSL).

Use prosuite.utils.get_ssl_channel_credentials() to create the basic https credentials if the appropria root certificates are in the windows certificate store. For advanced scenarios or credentials on a non-windows platform, see the gRPC Python docs (https://grpc.github.io/grpc/python/grpc.html).

verify(specification: Specification | XmlSpecification | DdxSpecification, perimeter: EnvelopePerimeter | EsriShapePerimeter | WkbPerimeter = None, output_dir: str = None, parameters: VerificationParameters = None) → Iterable[VerificationResponse]

Executes a quality verification by running all the quality conditions defined in the provided quality specification. Returns a collection of VerificationResponse objects, containing the verification messages.

Please refer to the samples for more details.

Parameters:

  • specification – The quality specification containing the conditions to be verified. It can be a prosuite.quality.Specification directly defined in python code, a prosuite.quality.DdxSpecification defined in the data dictionary or a prosuite.quality.XmlSpecification from an XML file, for example created by the XML export in the ProSuite Data Dictionary Editor.

  • perimeter – The perimeter that defines the polygon or extent of the verification. Default: None -> Full extent of the verified datasets.

  • output_dir – The output directory (must be writable / creatable by the service process). Default: No output is written by the server process.

  • parameters – Additional optional verification parameters.

Returns:

Iterator for looping over VerificationResponse objects. The verification response contains progress information, found issues and, in the final message, the verification results.

Return type:

Iterator[VerificationResponse]

class prosuite.verification.ServiceStatus

Bases: object

status_0 = 'Undefined'
status_1 = 'Running'
status_2 = 'Cancelled'
status_3 = 'Finished'
status_4 = 'Failed'
class prosuite.verification.VerificationParameters(tile_size: int = 5000, user_name: str = None)

Bases: object

Contains all parameters that can be passed to a verification.

add_objects_to_verify(dataset_id: int, object_ids: list)

Adds a dataset and a list of object IDs to the objects to be verified.

Parameters:

  • dataset_id (int) – The dataset ID containing the selected datasets The id can be look up in the DataDictionaryEditor -> Data -> Data Models -> Dataset -> Properties.

  • object_ids (list) – A list of feature-object IDs from the dataset to be verified.

desired_parallel_processing: int

The desired number of parallel worker processes to be used if the server allows parallel processing.

objects_to_verify

A dictionary containing the dataset IDs and the object IDs to be verified. The dataset ID can be looked up in the Data Dictionary Editor -> Data -> Data Models -> Dataset -> Properties. Use the method add_objects_to_verify to add a list of IDs for a specific dataset.

report_culture_code

The culture code that determines the language of the report. If None, the culture code of the machine of the server will be used. Examples: ‘de-CH’, ‘de-DE’, ‘de-AT’, ‘en-GB’

save_verification_statistics

If True, the verification statistics will be saved into the Data Dictionary database.

tile_size: int

The size (in meter) for testing quality conditions.

update_issues_in_verified_model

If True, the issues will be updated in the error datasets of the verified model.

user_name: str

The executing user which will be used in issue features.

class prosuite.verification.VerificationResponse(service_call_status: str, message: str, message_level: str, issue_msgs: list)

Bases: object

This class represents a VerificationResponse Message. The str() method is overridden to return all properties from the VerificationResponse when a VerificationResponse object is printed using pythons print() method.

issues
message: str

the actual message

message_level

message level -> see class MessageLevel

service_call_status

service status -> see class ServiceStatus

class prosuite.verification.WkbPerimeter(wkb: bytes)

Bases: object

A polygon in the OGC well-known-binary format. For example, in ArcPy the geometry’s WKB property could be used to acquire this format.

Utils

prosuite.utils.append_timestamp_to_basepath(base_path)
prosuite.utils.get_ssl_channel_credentials() → grpc.ChannelCredentials

Creates an ssl channel credentials object for use with an SSL-enabled channel for the authentication with a server that requires TLS/SSL. The appropriate root certificates for server authentication are loaded from the windows certificate store.

Returns:

A ChannelCredentials object

prosuite.utils.get_value_or_default(value: Any, default: Any) → Any

Returns the value if it is not None. Else returns the default value.

prosuite.utils.load_file_as_string(file_name: str) → str
prosuite.utils.load_json_file(json_file_path: str)
prosuite.utils.objectify_xml(path_file) → lxml.objectify.ObjectifiedElement

Try to objectify the xml file and return the objectified element. Return None if it fails.

prosuite.utils.to_bool(value: str | bool | int | None = None, default_value: bool = False) → bool

” treats “true” or “True” or ‘yes’ or ‘Yes’ or 1 or ‘1’ as True (bool). Anything else returns False (bool)

prosuite.utils.to_float(value: str | int | float | None, default_value: float = 0) → float | None
Parameters:

  • value – value that gets converted to float type

  • default_value – value that is returned in case the float conversion fails

Returns:

prosuite.utils.to_int(value: str | int | None, default_value: int = 0) → int | None

Tries to convert the input value to int. If conversion is not possible, returns the default value.

Parameters:

  • value – value that gets converted to int type

  • default_value – value that is returned in case the int conversion fails

Returns:

int representation of the input value or 0

prosuite.utils.to_spatial_reference_xml(sr_id: str)
prosuite.utils.try_get_from_oe(prop: str, object: lxml.objectify.ObjectifiedElement) → str | None
prosuite.utils.try_get_from_str_dict(key: str, dictionary: dict, default_value: int | float | str | None = '') → int | float | str | None

returns the value of the lookup element if it is available in the dict. if the lookup element is not in the dict, the default value is returned.

prosuite.utils.try_get_lxml_attrib(node: lxml.objectify.ObjectifiedElement, attribute_name: str) → str | None

If the attribute exists on the objectified node then return it, else return None

Organisational modules - Subpackages