Zilliz ups vector search results with Milvus database update

Byhassanhosting531@gmail.com

With vector indexing a crucial aspect of AI development, the latest release of Zilliz’s open source Milvus vector database is designed to improve the vector search results that feed AI pipelines.

Milvus 3.0, which was made generally available on July 16, features an architectural update that simplifies development by adding lake-native data access. Native access to data lakes such as Amazon S3, Google BigLake and Microsoft Azure Data Lake Storage enables Milvus to read, write and query data where it’s stored using open file formats including Apache Iceberg.

Previously, users had to extract and load data from data lakes into Milvus as part of their development pipelines, adding cost, complexity and data movement that could accidentally expose proprietary data.

In addition, Milvus now features an updated data retrieval engine to better discover and operationalize contextually appropriate vectors for agents and other AI applications that require large amounts of relevant data to deliver accurate results.

Given that using Milvus now eliminates certain data migration projects, the latest update is significant for not only existing users but also potential new ones, according to Mike Leone, an analyst at Moor Insights & Strategy.

“[It’s valuable] as long as your data already sits in one of the open table formats Milvus can read,” he told TechTarget. “If it does, adopting Milvus isn’t a migration project, and that opens it up to teams that could never justify the migration in the first place.”

Based in Redwood City, Calif., Zilliz is a vector database vendor that competes with fellow specialists such as Pinecone, Qdrant and Redis, as well as broader-based data management providers that offer vector database capabilities, including AWS, Google Cloud and Oracle.

In addition to Milvus, Zilliz offers Zilliz Cloud, a managed service that includes Milvus’ vector database capabilities.

Aiding AI development

As interest in building agents and other AI applications has surged, so has the importance of vector search.

Vectors are numerical representations of data that make information, including unstructured data such as text and audio, searchable so it can found and used to feed inform applications.

[It’s valuable] as long as your data already sits in one of the open table formats Milvus can read. If it does, adopting Milvus isn’t a migration project, and that opens it up to teams that could never justify the migration in the first place.
Mike LeoneAnalyst, Moor Insights & Strategy

As vector databases have gained widespread popularity over the past few years, providers have updated and improved their capabilities to deliver more accurate search results so that the data fed into AI pipelines enables organizations to build trustworthy AI tools.

For example, MongoDB and Teradata have each added new vector embedding and search capabilities this year.

Now, Zilliz is following suit with its latest Milvus database update, adding capabilities tailored to meet the needs of users doing more complex work than when they were building analytics and traditional AI tools such as predictive models, according to James Luan, co-founder and CTO of Zilliz.

“[The update] was very much driven by what we were hearing from production AI users, including large language model labs and self-driving companies,” he told TechTarget. “As AI workloads matured, teams weren’t just doing simple search anymore.”

One of the most pressing problems organizations faced was wasted time, effort and money spent copying and moving data from data lakes into Milvus, Luan continued.

“It’s the kind of invisible infrastructure tax that slows AI teams down considerably,” he said. “Milvus 3.0 tackles this by bringing production-grade vector indexes directly to lake-resident data. … Teams can search, analyze, and process from one copy, which is simpler to manage and much less expensive to operate.”

New Milvus database capabilities aimed at bringing vector search closer to where data is stored and improving the relevancy of vector search include the following:

  • Loon, a new storage engine that optimizes object storage reads to reduce latency and improve the accuracy of searches.
  • StructList, a multi-vector retrieval tool that enables users to discover documents, images and other data represented by more than one vector.
  • An index that strengthens sparse and hybrid vector retrieval searches while reducing the compute power required to run such searches.
  • External Collections, a feature that defines collections in Milvus over data stored in Apache Iceberg, Apache Parquet, Lance or Vortex to build searchable indexes.
  • Snapshots, a tool that enables users to create point-in-time, read-only views of collections so they can perform offline jobs such as evaluations and validations while production work continues in online environments.
  • A Spark connector that exposes Milvus as a data source for Amazon EMR, Databricks and Apache Spark pipelines as part of standard batch data workflows.

Kevin Petrie, an analyst at BARC U.S., noted that Zilliz’s blending of database and data lake capabilities in Milvus 3.0, which helps users more easily combine previously disparate analytical and transactional data workloads, plays to a growing trend.

Numerous vendors, including Databricks and Snowflake, are now providing database capabilities that enable customers to unify data to build a foundation for AI.

“Zilliz’s support of lakebase environments plays well to the convergence trend, which leading platforms such as Databricks are pushing aggressively,” Petrie told TechTarget. “Zilliz enables enterprises to deploy rich vector capabilities to support those converged workloads in lakebase environments.”

In addition, the Milvus update is significant for users given that it reduces the need to migrate data and improves the database’s search and indexing capabilities, he continued.

“Zilliz addresses several market requirements with this release,” Petrie said. “First, data remains highly distributed, [so] Zilliz is wise to help customers index vectors in place without moving them. … Second, enterprises need multimodal AI that goes beyond basic vectorization to handle capabilities such as the full-text search and indexing that Zilliz provides.”

Competitive standing

Although Zilliz is addressing user needs with its latest Milvus database update, the vendor faces a significant challenge as it competes for market share, according to Petrie.

With hyperscale cloud and data platform providers adding vector search and storage capabilities over the past few years, many enterprises that already use those providers for data management no longer need to seek out a specialist for their vector database needs. Specialists sometimes still provide more robust vector embedding and search capabilities than broader-based vendors, which appeals to certain potential customers. But for many, simplification is a selling point.

“Zilliz faces a similar challenge as all vector database companies, [which] is that vectors are just one piece of the multimodal puzzle,” Petrie said. “Larger vendors such as Google manage and retrieve vector, graph and tabular data from the same multimodal database, which simplifies enterprise AI architectures compared with point vector solutions.”

Leone noted that Snapshots is perhaps the most valuable of the new features because it allows users to evaluate their retrieval systems even if their data is continually changing.

From a competitive standpoint, he added that, while certain new Milvus capabilities are found in other vector databases as well, indexing data formats that aren’t converted to vectors — Apache Iceberg, Apache Parquet, Lance and Vortex — is a potential differentiator for Zilliz.

“The area to pay attention to is indexing formats you never had to convert first,” Leone said. “That’s harder than it sounds, because serving live queries straight off object storage is slow. It’s why Zilliz built Loon, the new storage engine underneath Milvus.”

Looking ahead

With the latest version of Milvus now GA, over the second half of 2026 Zilliz plans to add deeper integrations between the database and data lakes that better enable users to audit and debug AI behavior, according to Luan.

In addition, smarter storage and elasticity at scale to optimize usage are part of the product development roadmap for Milvus, he continued.

Beyond Milvus, Zilliz plans to bring all the new capabilities planned for its open source database to its Zilliz Cloud offering.

“Bringing the full Milvus 3.0 feature set into this managed platform is a key part of our [second half] roadmap,” Luan said.

Leone, meanwhile, suggested that Zilliz add security capabilities beyond access controls to reduce reliance on system administrators.

“If Milvus is reading straight from a customer’s tables, someone in security is going to ask who can see what,” he said. “Milvus already has its own access controls, so next I’d want the permissions on those tables to carry through automatically. Otherwise, an admin is keeping two separate systems in sync by hand.”

Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *