就是一个简单记录，基于s3 进行文件存储还是比较方便的
环境准备
- docker-compose.yaml
version: '3'
services:
minio:
image: minio/minio
ports:
- "9002:9000"
- "19001:19001"
environment:
MINIO_ACCESS_KEY: minio
MINIO_SECRET_KEY: minio123
command: server --console-address :19001 --quiet /data
- requirements.txt
主要是依赖，方便通过venv 安装
pandas
s3fs
fsspec
pyarrow
读取&效果
- s3app.py
import pandas as pd
parquet_file = 's3://demo/file2.parquet'
storage_conf = {'endpoint_url':'http://localhost:9002',"key":"minio","secret":"minio123"}
result = pd.read_parquet(parquet_file, engine='pyarrow',storage_options=storage_conf)
print(result.head())
- 效果
说明
pandas 对于s3 的读取依赖了fsspec，同时实际读取需要安装对应的s3 实现
参考资料
https://s3fs.readthedocs.io/en/latest/
https://filesystem-spec.readthedocs.io/en/latest/
https://pandas.pydata.org/docs/reference/api/pandas.read_parquet.html