# Task Queue Python API Overview - Google App Engine - Google Code

English ▼
Skip to page content Skip to main navigation
e.g. "templates" or "datastore"

# Task Queue Python API Overview

With the Task Queue API, applications can perform work outside of a user request, initiated by a user request. If an app needs to execute some background work, it can use the Task Queue API to organize that work into small, discrete units, called tasks. The app adds tasks to task queues to be executed later.

App Engine provides two different queue configurations:

1. Push queues process tasks based on the processing rate configured in the queue definition. App Engine automatically scales processing capacity to match your queue configuration and processing volume, and also deletes tasks after processing. Push queues are the default.
2. Pull queues allow a task consumer (either your application or code external to your application) to lease tasks at a specific time for processing within a specific timeframe. Pull queues give you more control over when tasks are processed, and also allow you to integrate your application with non-App-Engine code using the experimental Task Queue REST API. When using pull queues, your application needs to handle scaling of instances based on processing volume, and also needs to delete tasks after processing.

This page provides the basic concepts common to both types of queues. Once you've understood the basics, you can check out the queue configuration page, push queue overview, and pull queue overview to see how to configure and use these two types of queues.

- Queue Concepts
  - The Default Queue
  - Named Queues
  - Task Queues in the Administration Console
- Task Concepts
  - Task Names
  - Tasks within Transactions
  - Deleting Tasks
- Status of the Task Queue API

## Queue Concepts

Tasks queues are an efficient and powerful tool for background processing; they allow your application to define tasks, add them to a queue, and then use the queue to process them in aggregate. You name queues and configure their properties in a configuration file named `queue.yaml`. A queue can be one of two types—push or pull—based on your needs.

Push queues function only within the App Engine environment. These queues are the best choice for applications whose tasks work only with App Engine tools and services. With push queues, you simply configure a queue and add tasks to it. App Engine handles the rest. Push queues are easier to implement, but are restricted to use within App Engine. For more information about push queues and examples of how to use them, see Using Push Queues.

If you want to use a different system to consume tasks, pull queues are your best choice. In pull queues, a task consumer (either in your App Engine application, a backend, or code outside of App Engine) leases a specific number of tasks from a specific queue for a specific timeframe. After leasing tasks from a pull queue, the task consumer is responsible for deleting them. If you are consuming tasks from within App Engine, you can use calls from the `google.apphosting.api.taskqueue` module. If you are consuming tasks from outside of App Engine, you need to use the Task Queue REST API. Pull queues give you more power and flexibility over when and where tasks are processed, but they require you to handle scaling of workers based on processing volume. Your task consumer also needs to delete tasks after processing.

In summary, push queues allow you to process tasks within App Engine at a steady rate and App Engine scales computing resources according to the number of tasks in your queue. Pull queues allow an alternate task consumer to process tasks at a specific time, either in or outside App Engine, but your application needs to scale workers based on processing volume, as well as delete tasks after processing.

You can read more about these two types of queues in the Using Push Queues and Using Pull Queues.

### The Default Queue

For convenience, App Engine provides a default push queue to all applications (there is no default pull queue). If you do not name a queue for a task, App Engine automatically inserts it into the default queue. You can use this queue immediately without any additional configuration.

The default queue is preconfigured with a throughput rate of 5 task invocations per second. If you want to change the preconfigured settings, simply define a queue named `default` in `queue.yaml`. Code may always insert new tasks into the default queue, but if you wish to disable execution of these tasks, you may do so by clicking the Pause Queue button in the Task Queues tab of the Administration Console.

### Named Queues

While the default queue makes it easy to enqueue tasks with no configuration, you can also create custom queues by defining them in `queue.yaml`. Custom queues allow you to more effectively handle task processing by grouping similar types of tasks. You can control the processing rate—and a number of other settings—based specifically on the type of task in each queue.

For more information about configuring queues in `queue.yaml`, please see Python Task Queue Configuration.

### Task Queues in the Administration Console

You can manage task queues for an application using the Task Queue tab of the Administration Console. The Task Queue tab lists all of the queues in the application. Clicking on a queue name brings up the Task Queue Details page where you can see all of the tasks scheduled to run in a queue and you can manually delete individual tasks or purge every task from a queue. This is useful if a task in a push queue cannot be completed successfully and is stuck waiting to be retried. You can also pause and resume a queue on this page.

You can view details of individual tasks by clicking the task name from the list of tasks on the Task Queue Details page. This page allows you to debug why a task did not run successfully. You can also see information about the previous run of the task as well as the task's body.

Note: The default queue appears in the Console only after the app has enqueued a task to it for the first time.

## Task Concepts

A task is a unit of work to be performed by the application. Each task is an object of the Task class. Each Task object contains an endpoint (with a request handler for the task and an optional data payload that parameterizes the task). You can enqueue push tasks to a queue defined in `queue.yaml`. Push tasks and pull tasks are defined differently; see the Using Push Queues and Using Pull Queues for specific usage details.

### Task Names

In addition to a task's contents, you can declare a task's name. Once a task with name N is written, any subsequent attempts to insert a task named N fail.

While this is generally true, task names do not provide an absolute guarantee of once-only semantics. In rare cases, multiple calls to create a task of the same name may succeed. It's also possible in exceptional cases for a task to run more than once—even if it was only created once.

All project, queue, and task names must be a combination of one or more digits, letters a–z, underscores, and/or dashes, satisfying the following regular expression:

    [0-9a-zA-Z\-\_]+

If a push task is created successfully, it will eventually be deleted (at most seven days after the task successfully executes). Once deleted, its name can be reused.

If a pull task is created successfully, your application needs to delete the task after processing. The system may take up to seven days to recognize that a task has been deleted; during this time, the task name remains unavailable. Attempting to create another task during this time with the same name will result in an "item exists" error. The system offers no method to determine if deleted task names are still in the system. To avoid these issues, we recommend that you let App Engine generate the task name automatically.

### Tasks within Transactions

You can enqueue a task as part of a datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully. Tasks added within a transaction are considered to be a part of it and have the same level of isolation and consistency.

An application cannot insert more than five transactional tasks into task queues during a single transaction. Transactional tasks must not have user-specified names.

The following code sample demonstrates how to insert transactional tasks into a push queue as part of a datastore transaction:

    from google.appengine.api import taskqueue
    from google.appengine.ext import db

    def do_something_in_transaction():
      taskqueue.add(url='/path/to/my/worker', transactional=True)
      #...

    db.run_in_transaction(do_something_in_transaction)

### Deleting Tasks

You have three ways to delete tasks:

1. Individually, using the Administration Console.
2. Deleting all tasks for an entire queue, using the Administration Console.
3. Programmatically, using purge(), which deletes all tasks from the specified queue:

       from google.appengine.api import taskqueue

       q = taskqueue.Queue('queue1')
       q.purge()

   It takes the system about 20 seconds to notice that the queue has been purged. In push queues, tasks continue to execute during this time.

   Warning! Do not create new tasks in the same second as the purge call. Tasks created in close temporal proximity to a purge call will also be purged.

4. Programmatically, using delete_tasks() to delete individual tasks.

       from google.appengine.api import taskqueue

       q = taskqueue.Queue('queue1')
       q.delete_tasks(taskqueue.Task(name='foo'))

Note: It may take several hours to reclaim the quota used by queues purged programatically or through the Administration Console.

## Status of the Task Queue API

The App Engine Task Queue API is now available at its final location of:

`google.appengine.api.taskqueue`

The previous location of the App Engine Task Queue API is now deprecated. If you have used the App Engine Task Queue API while it was a labs feature, please update your code to remove any deprecation warnings.