This is a Docker image based on the Parquet CLI that ships with the Apache Parquet project. My fork of that project is at https://github.com/mneedham/parquet-mr/tree/master/parquet-cli
You can use this image to analyze Parquet images.
You can get the list of sub commands by calling the image:
docker run markhneedham/pq
For any sub-commands, you'll need to bind the directory containing your Parquet file so that the container can see it. e.g. if your Parquet file is in the data directory, you would do this:
docker -v $PWD/data:/data run markhneedham/pq <sub-command> /data/<file-name>.parquet
Show the schema and high-level overview of row groups.
docker run -v $PWD/data:/data markhneedham/pq meta \
/data/male_players_duck_1m.parquet
File path: /data/male_players_duck_1m.parquet
Created by: DuckDB
Properties: (none)
Schema:
message duckdb_schema {
optional binary player_id (STRING);
optional binary player_url (STRING);
optional int32 fifa_version (INTEGER(32,true));
optional int32 fifa_update (INTEGER(32,true));
optional int32 fifa_update_date (DATE);
...
}
Row group 0: count: 1001472 41.93 B records start: 4 total(compressed): 40.042 MB total(uncompressed):0 B
--------------------------------------------------------------------------------
type encodings count avg size nulls min / max
player_id BINARY Z RRR_ 1001472 0.40 B 20480 "102881" / "54033"
player_url BINARY Z RR_ 1001472 4.12 B 0 "/player/102881/kyriakos-s..." / "/player/54033/tom-huddles..."
fifa_version INT32 Z RR_ 1001472 0.00 B 0 "22" / "23"
fifa_update INT32 Z RR_ 1001472 0.00 B 0 "1" / "64"
fifa_update_date INT32 Z RR_ 1001472 0.00 B 0 "2021-12-13" / "2023-01-13"
short_name BINARY Z RRR_ 1001472 0.50 B 0 "A Lan" / "Ș. Vlădoiu"
...
Show the first few rows in a Parquet file
docker run -v $PWD/data:/data markhneedham/pq head -n 1 \
/data/male_players_duck_1m.parquet
{"player_id": null, "player_url": "/player/158023/lionel-messi/230009", "fifa_version": 23, "fifa_update": 9, "fifa_update_date": 19370, "short_name": "L. Messi", "long_name": "Lionel Andrés Messi Cuccittini", "player_positions": "RW", "overall": 91, "potential": 91, "value_eur": 54000000, "wage_eur": 195000, "age": 35, "dob": 6383, "height_cm": 169, "weight_kg": 67, "league_id": 16, "league_name": "Ligue 1", "league_level": 1, "club_team_id": 73, "club_name": "Paris Saint Germain", "club_position": "RS", "club_jersey_number": 30, "club_loaned_from": null, "club_joined_date": 18849, "club_contract_valid_until_year": 2023, "nationality_id": 52, "nationality_name": "Argentina", "nation_team_id": 1369, "nation_position": "RW", "nation_jersey_number": 10, "preferred_foot": "Left", "weak_foot": 4, "skill_moves": 4, "international_reputation": 5, "work_rate": "Low/Low", "body_type": "Unique", "real_face": "Yes", "release_clause_eur": 99900000, "player_tags": "#Dribbler, #Distance Shooter, #FK Specialist, #Acrobat, #Clinical Finisher, #Complete Forward", "player_traits": "Finesse Shot, Long Shot Taker (AI), Playmaker (AI), Outside Foot Shot, Chip Shot (AI), Technical Dribbler (AI)", "pace": 81, "shooting": 89, "passing": 90, "dribbling": 94, "defending": 34, "physic": 64, "attacking_crossing": 84, "attacking_finishing": 90, "attacking_heading_accuracy": 70, "attacking_short_passing": 91, "attacking_volleys": 88, "skill_dribbling": 95, "skill_curve": 93, "skill_fk_accuracy": 93, "skill_long_passing": 90, "skill_ball_control": 93, "movement_acceleration": 87, "movement_sprint_speed": 76, "movement_agility": 91, "movement_reactions": 92, "movement_balance": 95, "power_shot_power": 86, "power_jumping": 68, "power_stamina": 70, "power_strength": 68, "power_long_shots": 91, "mentality_aggression": 44, "mentality_interceptions": 40, "mentality_positioning": 93, "mentality_vision": 94, "mentality_penalties": 75, "mentality_composure": 96, "defending_marking_awareness": 20, "defending_standing_tackle": 35, "defending_sliding_tackle": 24, "goalkeeping_diving": 6, "goalkeeping_handling": 11, "goalkeeping_kicking": 15, "goalkeeping_positioning": 14, "goalkeeping_reflexes": 8, "goalkeeping_speed": null, "ls": "87+3", "st": "87+3", "rs": "87+3", "lw": "90", "lf": "91", "cf": "91", "rf": "91", "rw": "90", "lam": "92-1", "cam": "92-1", "ram": "92-1", "lm": "89+2", "lcm": "85+3", "cm": "85+3", "rcm": "85+3", "rm": "89+2", "lwb": "64+3", "ldm": "63+3", "cdm": "63+3", "rdm": "63+3", "rwb": "64+3", "lb": "59+3", "lcb": "50+3", "cb": "50+3", "rcb": "50+3", "rb": "59+3", "gk": "19+3", "player_face_url": "https://cdn.sofifa.net/players/158/023/23_120.png"}
Show the pages for all columns or a specified column.
docker run -v $PWD/data:/data markhneedham/pq pages -c player_id \
/data/male_players_duck_1m.parquet
Column: player_id
--------------------------------------------------------------------------------
page type enc count avg size size rows nulls min / max
0-D dict Z _ 27248 10.00 B 266.052 kB
0-1 data Z R 1001472 2.94 B 2.807 MB
1-D dict Z _ 25361 10.00 B 247.613 kB
1-1 data Z R 1001472 3.00 B 2.865 MB
2-D dict Z _ 26116 10.00 B 254.949 kB
2-1 data Z R 1001472 3.00 B 2.865 MB
3-D dict Z _ 24680 9.99 B 240.850 kB
3-1 data Z R 1001472 3.00 B 2.865 MB
4-D dict Z _ 24805 9.99 B 241.995 kB
4-1 data Z R 1001472 3.00 B 2.865 MB
5-D dict Z _ 21023 9.99 B 205.037 kB
5-1 data Z R 1001472 3.00 B 2.865 MB
6-D dict Z _ 23005 9.98 B 224.286 kB
6-1 data Z R 1001472 3.00 B 2.865 MB
7-D dict Z _ 20333 9.98 B 198.160 kB
7-1 data Z R 1001472 3.00 B 2.865 MB
8-D dict Z _ 24456 9.98 B 238.243 kB
8-1 data Z R 1001472 3.00 B 2.865 MB
9-D dict Z _ 23827 9.97 B 231.885 kB
9-1 data Z R 990342 3.00 B 2.833 MB
Content type
Image
Digest
sha256:0ee651fb1…
Size
326.2 MB
Last updated
over 3 years ago
docker pull markhneedham/pq