This blog series captures practical learnings from working with GPUs in real‑world environments, with a focus on operations, reliability, and scale. Each post deep‑dives into specific aspects of GPU systems based on hands‑on experience, incidents, and operational challenges. Together, these articles aim to share actionable insights, highlight common pitfalls, and help teams build more robust and predictable GPU operations.
Azure AI Foundry supports deploying large language models (LLMs). In this article, we will see how to deploy a model and use it.
Azure AI Foundry is a comprehensive suite of tools and services designed to accelerate the development and deployment of AI solutions on the Azure platform. Throughout this blog series, we will cover various aspects of Azure AI Foundry.
In this article, I will show you how to use Azure Cognitive Services for text summarization.
Hope this was useful. Cheers!
In this article, I will show you how to use an Azure AI Service available within the Azure AI Foundry project. We'll use the language translator as an example.
import os, requests, uuid, json
resource_key = 'resource_key_here'
region = 'region_here'
endpoint = 'https://api.cognitive.microsofttranslator.com/'
# If you encounter any issues with the base_url or path, make sure
# that you are using the latest endpoint: https://docs.microsoft.com/azure/cognitive-services/translator/reference/v3-0-translate
path = '/translate?api-version=3.0'
params = '&to=ml'
constructed_url = endpoint + path + params
headers = {
'Ocp-Apim-Subscription-Key': resource_key,
'Ocp-Apim-Subscription-Region': region,
'Content-type': 'application/json',
'X-ClientTraceId': str(uuid.uuid4())
}
# You can pass more than one object in body.
body = [{
'text' : 'where are you right now?'
}]
request = requests.post(constructed_url, headers=headers, json=body)
response = request.json()
print(json.dumps(response, sort_keys=True, indent=4, separators=(',', ': ')))
Sample output:
What you see is a unicode string and once it is converted you will see the corresponding Malayalam text.
curl -X POST "https://api.cognitive.microsofttranslator.com/translate?api-version=3.0&to=ml" \
> -H "Ocp-Apim-Subscription-Key: your_key_here" \
> -H "Ocp-Apim-Subscription-Region: your_region_here" \
> -H "Content-Type: application/json" \
> -d "[{'Text':'where are you now?'}]" -v
Sample output:
Hope this was useful. Cheers!
In this blog series we will explore the fascinating world of Generative AI and Large Language Models (LLMs). We delve into the latest advancements in AI technology, focusing particularly on LLMs, which have revolutionized various fields, including natural language processing and text generation.
Throughout this series, we will discuss LLM serving platforms such as Ollama and Hugging Face, providing insights into their capabilities, features, and applications. I will also guide you through the process of getting started with LLMs, from setting up your development/ test environment to deploying these powerful models on Kubernetes clusters. Additionally, we'll demonstrate how to effectively prompt and interact with LLMs using frameworks like LangChain, empowering you to harness the full potential of these cutting-edge technologies.
Stay tuned for insightful articles, and hands-on guides that will equip you with the knowledge and skills to unlock the transformative capabilities of LLMs. Let's explore the future of AI together!
.jpeg) |
| Image credits: designer.microsoft.com/image-creator |
Part1 - Deploy Ollama on Kubernetes
Part2 - Prompt LLMs using Ollama, LangChain, and Python
Part3 - Web UI for Ollama to interact with LLMs
Part4 - Vision assistant using LLaVA
Part1 - Getting started with Hugging Face
Part2 - Code generation with Code Llama Instruct
Part3 - Inference with Code Llama using LangChain
Part4 - Containerize your LLM app using Python, FastAPI, and Docker
Part5 - Deploy your LLM app on Kubernetes
Part6 - LLM app observability <coming soon>
Docker published GenAI stack around Oct 2023 which consists of large language models (LLMs) from Ollama, vector and graph databases from Neo4j, and the LangChain framework. These utilities can help developers with the resources they need to kick-start creating new applications using generative AI. Ollama can be used to deploy and run LLMs locally. In this exercise we will deploy Ollama to a Kubernetes cluster and prompt it.
In my case I am using a Tanzu Kubernetes Cluster (TKC) running on vSphere with Tanzu 7u3 platform powered by Dell PowerEdge R640 servers. The TKC nodes are using best-effort-2xlarge vmclass with 8 CPU and 64Gi Memory. Note that I am running it on a regular Kubernetes cluster without GPU. If you have GPU, additional configuration steps might be required.
https://github.com/vineethac/Ollama/tree/main/ollama_on_kubernetes
