core.dispatcher.task.tasks

Functions

get_pipeline_options(job_name[, runner, …])

Build dispatcher options for profile_api.

get_task_logger(name)

Get logger for task module by name.

Classes

AutoModelForTokenClassification()

This is a generic model class that will be instantiated as one of the model classes of the library—with a token classification head—when created with the from_pretrained() class method or the from_config() class method.

AutoTokenizer()

This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the AutoTokenizer.from_pretrained() class method.

AzureBlobFileDownloader()

Celery([main, loader, backend, amqp, …])

Celery application.

PipelineOptions(flags, **kwargs)

This class and subclasses are used as containers for command line options.

Predict(modelName)

Loads the pretrained model which is then used for predicting values Can read more about it here: https://github.com/huggingface/transformers

Tokenize(model_name)

The tokenizer is responsible for all the preprocessing the pretrained model expects, and can be called directly on one (or list) of texts.
core.dispatcher.task core.dispatcher.test