Dwayo DevBlog

Tag: deployment

  • Gunicorn: FastAPI in Production

    1. Gunicorn Configuration

    Gunicorn is a widely used WSGI server for running Python web applications. When deploying FastAPI, Gunicorn is often used in conjunction with Uvicorn to provide a production-ready server.

    Install Gunicorn using:

    pip install gunicorn

    Run Gunicorn with Uvicorn workers:

    gunicorn -k uvicorn.workers.UvicornWorker your_app:app -w 4 -b 0.0.0.0:8000

    Here:

    • -k uvicorn.workers.UvicornWorker specifies the worker class.
    • your_app:app points to your FastAPI application instance.
    • -w 4 sets the number of worker processes. Adjust this based on the available resources and expected load.

    2. Worker Processes

    The -w flag in the Gunicorn command determines the number of worker processes. The optimal number depends on factors like CPU cores, available memory, and the nature of your application.

    For example, on a machine with four CPU cores:

    gunicorn -k uvicorn.workers.UvicornWorker your_app:app -w 4 -b 0.0.0.0:8000

    If your application performs a significant amount of asynchronous I/O operations, you might increase the number of workers. However, keep in mind that too many workers can lead to resource contention.

    3. Load Balancing and Scaling

    In a production setting, deploying multiple instances of your FastAPI application and distributing incoming requests across them is essential for scalability and fault tolerance. The number of worker processes can impact the optimal scaling strategy.

    Consider using tools like nginx for load balancing or deploying your application in a container orchestration system like Kubernetes.

    4. Graceful Shutdown

    Ensure that Gunicorn handles signals gracefully. FastAPI applications may have asynchronous tasks or background jobs that need to complete before shutting down. Gunicorn’s --graceful-timeout option can be set to allow for graceful termination.

    gunicorn -k uvicorn.workers.UvicornWorker your_app:app -w 4 -b 0.0.0.0:8000 --graceful-timeout 60

    This allows Gunicorn to wait up to 60 seconds for workers to finish processing before shutting down.

    In conclusion, the choice of Gunicorn and worker processes is a crucial aspect of deploying FastAPI applications in a production environment. Fine-tuning the number of workers and configuring Gunicorn parameters according to your application’s characteristics and deployment environment ensures optimal performance and scalability.

  • Deploying Your AI Model

    Once you have a trained machine learning model, exposing it as an API allows other applications to interact with and use the model’s predictions. Here’s a step-by-step guide on how to create an API for a trained model:

    1. Choose a Framework:

    • Decide on a web framework or technology to serve your model as an API. Flask (Python), Django, FastAPI, or TensorFlow Serving are common choices.

    2. Create a Web Server:

    • Use the chosen framework to set up a web server. For example, if you’re using Flask, you’d create a Flask application.
    from flask import Flask, request, jsonify
app = Flask(__name__)

    3. Load the Trained Model:

    • Load your pre-trained machine learning model into your application. This could be a TensorFlow or PyTorch model, for instance.
    # Example for loading a TensorFlow model in Flask
from tensorflow import keras
model = keras.models.load_model('path/to/your/model')

    4. Define an API Endpoint:

    • Create an endpoint that will receive input data and return model predictions. This is the function that will be called when someone queries your API.
    @app.route('/predict', methods=['POST'])
def predict():
    data = request.json  # Assuming JSON data is sent in the request
    # Perform any necessary preprocessing on the input data
    predictions = model.predict(data)
    # Format the predictions as needed
    return jsonify({'predictions': predictions.tolist()})

    5. Handle Input and Output:

    • Define how your API will handle input data and format the output. This includes any necessary data validation and post-processing steps.

    6. Run the Web Server:

    • Start the web server to make your API accessible. Depending on your chosen framework, this might involve running a command like flask run.

    7. Test Locally:

    • Test your API locally to ensure that it’s working as expected. You can use tools like curl or Postman to send requests and receive predictions.

    8. Deploy the API:

    • Choose a platform to deploy your API. This could be a cloud platform like AWS, Google Cloud, or Azure. Alternatively, you can deploy it on your own server.

    9. Expose the API:

    • Once deployed, expose your API to the internet. This might involve setting up a domain name and configuring security settings.

    10. Documentation:

    • Create documentation that explains how to use your API, including the expected input format, available endpoints, and how to interpret the output.