LLM-Alibaba Cloud DashVector ModelScope Multi-modalis vectorization real-time text search practical summary

LLM-Alibaba Cloud DashVector ModelScope Multi-modalis vectorization real-time text image search summary practical

2024-07-12

Articuli directorium

Praefatio
step
- Imago data Embedding reponitur in database
- Textus retrieval
Integram codice

Praefatio

Hoc articulo utitur Alibaba vector retrievalis nubis (DashVector), cum Unum-Pax exemplar multimodal , aedificare reales-time multimodis retrievalium capacitatum "textu inquisitionis pro imaginibus". Altiore processu talis est:

Multimoda notitia repono Embedding .Servientes per exemplar unum pacemEmbeddingInterfacius convertitur notitia ex pluribus modis in vectores summus dimensiva.
Multimoda Query retrieval . Fundatur ex multimodis Embedding facultatibus, exemplar unum-Pax provisum, libere possumus miscere varias initationes modales, ut unum textum, textum + audio, picturam audio + et alia multi-modalia initibus per DashVector simili modo transire potest.

PRAEREQUISITIS

Exemplar apertum Lingji servitium et API-KEY obtine;Activate DashScope et crea API-KEY
Aperta vector retrieval servitium: videSubscribe ut servitium。
Create vector retrieval servitium API-KEY: seeAPI-KEY procuratio。

Praeparatio Environmental

# 安装 dashscope 和 dashvector sdk
pip3 install dashscope dashvector

# 显示图片
pip3 install Pillow
1
2
3
4
5

notitia praeparatio

illustrare
Cum DashScope exemplar UNUM-PAX servitium currently tantum sustinet imaginem et audio inputationem in forma URLs, necesse est ut indices tabularum retis publici appositas (ut oss/s3) in antecessum obtineat ac domicilium inscriptionis indice respondentis imagines et audio.

step

Imago data Embedding reponitur in database

Alibaba Cloud OSS ad imaginem servandi usus sum, et per interfaciem OSS navigatoris imaginis domicilium exterius pervium consecutus sum:

Hoc domicilium etiam per interface obtineri debet. Hoc nondum investigatum est. Amici interest in batches per interfaciem impetrare possunt. Propositum hoc domicilium obtinendum est ut Alibaba Cloud's DashScope ministerium imagini embedding legendi possit. Salvum facere DashVector vector database.
Domicilio obtenta, URL scribe in nostro imagenet1k-urls.txt fasciculus, codice nostro tabellam postea emendandi leget:

Codicem ad embeddingum faciendum hoc modo est (postea dabo integram codicem ac structuram presul postea, modo in codice infixo hic ponitur);

    def index_image(self):

        # 创建集合：指定集合名称和向量维度, ONE-PEACE 模型产生的向量统一为 1536 维
        collection = self.vector_client.get(self.vector_collection_name)
        if not collection:
            rsp = self.vector_client.create(self.vector_collection_name, 1536)
            collection = self.vector_client.get(self.vector_collection_name)
            if not rsp:
                raise DashVectorException(rsp.code, reason=rsp.message)

        # 调用 dashscope ONE-PEACE 模型生成图片 Embedding，并插入 dashvector
        with open(self.IMAGENET1K_URLS_FILE_PATH, 'r') as file:
            for i, line in enumerate(file):
                url = line.strip('n')
                input = [{'image': url}]
                result = MultiModalEmbedding.call(model=MultiModalEmbedding.Models.multimodal_embedding_one_peace_v1,
                                                  input=input,
                                                  api_key=os.environ["DASHSCOPE_API_KEY"],
                                                  auto_truncation=True)
                if result.status_code != 200:
                    print(f"ONE-PEACE failed to generate embedding of {url}, result: {result}")
                    continue
                embedding = result.output["embedding"]
                collection.insert(
                    Doc(
                        id=str(i),
                        vector=embedding,
                        fields={'image_url': url}
                    )
                )
                if (i + 1) % 100 == 0:
                    print(f"---- Succeeded to insert {i + 1} image embeddings")

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

legere IMAGENET1K_URLS_FILE_PATHImago URL in Domicilio, et DashScope postulationem praestare ut vectorizet imaginemque nostram recondas.
Cum vector datorum inserendo, Domicilium imaginis attributum est ut vector.

Post supplicium transire potesVector retrieval servitium consolatoriumVectorem data reprehendo:

Textus retrieval

Ut notitias ex vectore database per textum retrieve, ingrediorcatPost tres tabulas receptas (top=3 in nostro codice positas), effectum inspicere potes. Duae imagines felium sunt, sed una canis imago;

Causa est, quia similes sunt canibus et felibustopkPone 2, theoretice canis deprehendi non potest. Videamus effectum, et certe satis est, canis non est;

Causa quare canes apparent, est quia picturas bestiarum 4 in bibliotheca vectoris reposui, picturas 2 cat et 2 caninas picturas.

Integram codice

multi_model.pyDocumenta haec sunt:

import os

import dashscope
from dashvector import Client, Doc, DashVectorException
from dashscope import MultiModalEmbedding
from dashvector import Client
from urllib.request import urlopen
from PIL import Image


class DashVectorMultiModel:
    def __init__(self):
        # 我们需要同时开通 DASHSCOPE_API_KEY 和 DASHVECTOR_API_KEY
        os.environ["DASHSCOPE_API_KEY"] = ""
        os.environ["DASHVECTOR_API_KEY"] = ""
        os.environ["DASHVECTOR_ENDPOINT"] = ""

        dashscope.api_key = os.environ["DASHSCOPE_API_KEY"]

        # 由于 ONE-PEACE 模型服务当前只支持 url 形式的图片、音频输入，因此用户需要将数据集提前上传到
        # 公共网络存储（例如 oss/s3），并获取对应图片、音频的 url 列表。
        # 该文件每行存储数据集单张图片的公共 url，与当前python脚本位于同目录下
        self.IMAGENET1K_URLS_FILE_PATH = "imagenet1k-urls.txt"

        self.vector_client = self.init_vector_client()
        self.vector_collection_name = 'imagenet1k_val_embedding'

    def init_vector_client(self):
        return Client(
          api_key=os.environ["DASHVECTOR_API_KEY"],
          endpoint=os.environ["DASHVECTOR_ENDPOINT"]
        )

    def index_image(self):

        # 创建集合：指定集合名称和向量维度, ONE-PEACE 模型产生的向量统一为 1536 维
        collection = self.vector_client.get(self.vector_collection_name)
        if not collection:
            rsp = self.vector_client.create(self.vector_collection_name, 1536)
            collection = self.vector_client.get(self.vector_collection_name)
            if not rsp:
                raise DashVectorException(rsp.code, reason=rsp.message)

        # 调用 dashscope ONE-PEACE 模型生成图片 Embedding，并插入 dashvector
        with open(self.IMAGENET1K_URLS_FILE_PATH, 'r') as file:
            for i, line in enumerate(file):
                url = line.strip('n')
                input = [{'image': url}]
                result = MultiModalEmbedding.call(model=MultiModalEmbedding.Models.multimodal_embedding_one_peace_v1,
                                                  input=input,
                                                  api_key=os.environ["DASHSCOPE_API_KEY"],
                                                  auto_truncation=True)
                if result.status_code != 200:
                    print(f"ONE-PEACE failed to generate embedding of {url}, result: {result}")
                    continue
                embedding = result.output["embedding"]
                collection.insert(
                    Doc(
                        id=str(i),
                        vector=embedding,
                        fields={'image_url': url}
                    )
                )
                if (i + 1) % 100 == 0:
                    print(f"---- Succeeded to insert {i + 1} image embeddings")

    def show_image(self, image_list):
        for img in image_list:
            # 注意：show() 函数在 Linux 服务器上可能需要安装必要的图像浏览器组件才生效
            # 建议在支持 jupyter notebook 的服务器上运行该代码
            img.show()

    def text_search(self, input_text):
        # 获取上述入库的集合
        collection = self.vector_client.get('imagenet1k_val_embedding')

        # 获取文本 query 的 Embedding 向量
        input = [{'text': input_text}]
        result = MultiModalEmbedding.call(model=MultiModalEmbedding.Models.multimodal_embedding_one_peace_v1,
                                          input=input,
                                          api_key=os.environ["DASHSCOPE_API_KEY"],
                                          auto_truncation=True)
        if result.status_code != 200:
            raise Exception(f"ONE-PEACE failed to generate embedding of {input}, result: {result}")
        text_vector = result.output["embedding"]

        # DashVector 向量检索
        rsp = collection.query(text_vector, topk=2)
        image_list = list()
        for doc in rsp:
            img_url = doc.fields['image_url']
            img = Image.open(urlopen(img_url))
            image_list.append(img)
        return image_list


if __name__ == '__main__':
    a = DashVectorMultiModel()
    # 执行 embedding 操作
    a.index_image()
    # 文本检索
    text_query = "Traffic light"
    a.show_image(a.text_search(text_query))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

Aperi API clavem DashScope et DashVector ac reponere supraDASHSCOPE_API_KEY,DASHVECTOR_API_KEY,DASHVECTOR_ENDPOINT

Codicis compages directorii talis est. Pone tabellam fasciculi et py fasciculi in eodem directorio:

Additional information

Utere loci imagines: Imaginem ad OSS imposuisti. Potes etiam imaginem imaginis localis uti ac tabellam iter in txt cum loci imaginis iter reponere, hoc modo:

Si localibus imaginibus utimur, codicem supra et codicem infra mutamus;

# 将 img = Image.open(urlopen(img_url)) 替换为下边的代码
img = Image.open(img_url)
1
2

Technology sharing