Skip to content

Commit 7cd29b8

Browse files
alingseclaude
alingse
and
claude
authored
Modernize to Python 3 and upgrade project infrastructure (#50)
* Modernize to Python 3 and upgrade project infrastructure - Remove Python 2 compatibility code (PY2, future imports, unicode literals) - Replace unicodecsv with Python 3 built-in csv module - Migrate from setup.py to pyproject.toml with hatchling build backend - Replace Travis CI with GitHub Actions (Python 3.8-3.13) - Convert README.rst to README.md - Add ruff for linting and formatting - Add mypy for type checking - Add pre-commit hooks configuration - Fix file handling to use context managers (with statements) - Update test files for Python 3 compatibility All 31 tests pass. Ruff checks pass. MyPy type checks pass. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix CI: add --system flag to uv pip install In GitHub Actions environment, uv requires --system flag when no virtual environment is present. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use uv run for consistent command execution Simplify CI by using 'uv run' instead of activating virtual environment. This keeps commands consistent with local development. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix CI: restore uv venv step and add cache - Add back 'uv venv' step that was accidentally removed - Add GitHub Actions cache for .venv directory - Cache key includes Python version and dependency hashes Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent d3ed43d commit 7cd29b8

19 files changed

Lines changed: 1416 additions & 428 deletions

‎.github/workflows/ci.yml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [master, main]
6+
pull_request:
7+
branches: [master, main]
8+
workflow_dispatch:
9+
10+
jobs:
11+
lint:
12+
name: Lint
13+
runs-on: ubuntu-latest
14+
steps:
15+
- uses: actions/checkout@v4
16+
- uses: actions/setup-python@v5
17+
with:
18+
python-version: "3.12"
19+
- name: Install uv
20+
run: |
21+
curl -LsSf https://astral.sh/uv/install.sh | sh
22+
echo "$HOME/.local/bin" >> $GITHUB_PATH
23+
- name: Set up UV cache
24+
uses: actions/cache@v4
25+
with:
26+
path: .venv
27+
key: uv-${{ runner.os }}-lint-${{ hashFiles('pyproject.toml', 'requirements.txt') }}
28+
restore-keys: |
29+
uv-${{ runner.os }}-lint-
30+
- name: Create virtual environment
31+
run: uv venv || true
32+
- name: Install dependencies
33+
run: uv pip install -e ".[dev]"
34+
- name: Run ruff
35+
run: uv run ruff check .
36+
- name: Run mypy
37+
run: uv run mypy jsoncsv
38+
39+
test:
40+
name: Test (Python ${{ matrix.python-version }})
41+
runs-on: ubuntu-latest
42+
strategy:
43+
matrix:
44+
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]
45+
steps:
46+
- uses: actions/checkout@v4
47+
- uses: actions/setup-python@v5
48+
with:
49+
python-version: ${{ matrix.python-version }}
50+
- name: Install uv
51+
run: |
52+
curl -LsSf https://astral.sh/uv/install.sh | sh
53+
echo "$HOME/.local/bin" >> $GITHUB_PATH
54+
- name: Set up UV cache
55+
uses: actions/cache@v4
56+
with:
57+
path: .venv
58+
key: uv-${{ runner.os }}-py${{ matrix.python-version }}-${{ hashFiles('pyproject.toml', 'requirements.txt') }}
59+
restore-keys: |
60+
uv-${{ runner.os }}-py${{ matrix.python-version }}-
61+
- name: Create virtual environment
62+
run: uv venv || true
63+
- name: Install dependencies
64+
run: uv pip install -e ".[dev]"
65+
- name: Run tests
66+
run: uv run pytest --cov=jsoncsv --cov-report=xml
67+
- name: Upload coverage
68+
uses: codecov/codecov-action@v4
69+
with:
70+
file: ./coverage.xml
71+
fail_ci_if_error: false
72+
73+
build:
74+
name: Build package
75+
runs-on: ubuntu-latest
76+
steps:
77+
- uses: actions/checkout@v4
78+
- uses: actions/setup-python@v5
79+
with:
80+
python-version: "3.12"
81+
- name: Install uv
82+
run: |
83+
curl -LsSf https://astral.sh/uv/install.sh | sh
84+
echo "$HOME/.local/bin" >> $GITHUB_PATH
85+
- name: Set up UV cache
86+
uses: actions/cache@v4
87+
with:
88+
path: .venv
89+
key: uv-${{ runner.os }}-build-${{ hashFiles('pyproject.toml') }}
90+
restore-keys: |
91+
uv-${{ runner.os }}-build-
92+
- name: Create virtual environment
93+
run: uv venv || true
94+
- name: Build package
95+
run: |
96+
uv pip install build twine
97+
uv run python -m build
98+
- name: Check package
99+
run: uv run twine check dist/*

‎.pre-commit-config.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Pre-commit configuration for jsoncsv
2+
# Install: pip install pre-commit
3+
# Run: pre-commit install
4+
5+
repos:
6+
- repo: https://github.com/astral-sh/ruff-pre-commit
7+
rev: v0.9.0
8+
hooks:
9+
- id: ruff
10+
args: [--fix]
11+
- id: ruff-format
12+
13+
- repo: https://github.com/pre-commit/mirrors-mypy
14+
rev: v1.14.1
15+
hooks:
16+
- id: mypy
17+
additional_dependencies:
18+
- click
19+
- xlwt
20+
args: [--ignore-missing-imports]

‎.travis.yml‎

Lines changed: 0 additions & 28 deletions
This file was deleted.

‎README.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
# jsoncsv : easily convert json to csv or xls[x]
2+
3+
[![PyPI](https://img.shields.io/pypi/v/jsoncsv.svg)](https://pypi.python.org/pypi/jsoncsv)
4+
[![CI](https://github.com/alingse/jsoncsv/actions/workflows/ci.yml/badge.svg)](https://github.com/alingse/jsoncsv/actions/workflows/ci.yml)
5+
[![Coverage](https://codecov.io/gh/alingse/jsoncsv/badge.svg)](https://codecov.io/gh/alingse/jsoncsv)
6+
[![Ruff](https://img.shields.io/endpoint?url=https://ruff.ai/api/badge)](https://github.com/astral-sh/ruff)
7+
8+
jsoncsv (with `mkexcel`) is a command tool to convert json file to csv/xlsx file.
9+
10+
It's simple, and no need user to specify the keys.
11+
12+
Just use them.
13+
14+
## Requirements
15+
16+
- Python 3.8 or higher
17+
18+
## Quick Start
19+
20+
Cat the raw.json to csv/xls use command line tool
21+
22+
```bash
23+
cat raw.json | jsoncsv | mkexcel > output.csv
24+
cat raw.json | jsoncsv | mkexcel -t xls > output.xls
25+
```
26+
27+
Each line of raw json text file is a json object
28+
29+
```json
30+
{"id":1, "name":"A", "year": 2015}
31+
{"id":2, "name":"S", "zone": "china"}
32+
```
33+
34+
Jsoncsv will output a csv file
35+
36+
```
37+
id,name,year,zone
38+
1,A,2015,
39+
2,S,,china
40+
```
41+
42+
This is easily and needn't care the different keys from any two object.
43+
44+
### For Json Array
45+
46+
If input file is an json_array, use `-A/--array` to decode it
47+
48+
```bash
49+
cat raw.json | jsoncsv -A | mkexcel > output.csv
50+
```
51+
52+
**Input File**
53+
54+
```json
55+
[
56+
{"id":1, "name":"A", "year": 2015},
57+
{"id":2, "name":"S", "zone": "china"}
58+
]
59+
```
60+
61+
**Output File**
62+
63+
```
64+
id,name,year,zone
65+
1,A,2015,
66+
2,S,,china
67+
```
68+
69+
### Step by Step
70+
71+
```bash
72+
jsoncsv raw.json expand.json
73+
mkexcel expand.json -t xls output.xls
74+
```
75+
76+
get more options with `--help`.
77+
78+
```bash
79+
jsoncsv --help
80+
mkexcel --help
81+
```
82+
83+
## Install
84+
85+
Using pip (recommended):
86+
87+
```bash
88+
pip install jsoncsv
89+
```
90+
91+
Using uv (faster):
92+
93+
```bash
94+
uv pip install jsoncsv
95+
```
96+
97+
For development:
98+
99+
```bash
100+
git clone https://github.com/alingse/jsoncsv
101+
cd jsoncsv
102+
uv pip install -e ".[dev]"
103+
```
104+
105+
## Usage
106+
107+
### Expand/Restore JSON
108+
109+
Just expand/restore the json, the expand json is one layer json.
110+
111+
```bash
112+
jsoncsv raw.json expand.json
113+
jsoncsv -r expand.json raw.json
114+
cat raw.json | jsoncsv | jsoncsv -r > raw2.json
115+
```
116+
117+
### Export to Excel/CSV
118+
119+
mkexcel the expanded json (one layer)
120+
121+
```bash
122+
mkexcel expand.json output.csv
123+
mkexcel -t xls expand.json > output.xls
124+
mkexcel -t csv expand.json > output.csv
125+
```
126+
127+
## Options
128+
129+
### -e, --expand
130+
131+
expand json, 展开 json
132+
133+
```bash
134+
jsoncsv -e raw.json expand.json
135+
cat raw.json expand.json
136+
{"s":[1,2,{"w":1}]}
137+
{"s.2.w": 1,"s.0": 1,"s.1": 2}
138+
```
139+
140+
`{"s":[1,2,{"w":1}]}` will transformed to `{"s.2.w": 1,"s.0": 1,"s.1": 2}`
141+
142+
the output "expand.json" is only one layer json, it can be easy change to csv or xlsx (with `mkexcel`)
143+
144+
### -r, --restore
145+
146+
restore the expanded json 重构被展开的json
147+
148+
```bash
149+
jsoncsv -r expand.json raw.json
150+
cat expand.json raw.json
151+
{"s.2.w": 1,"s.0": 1,"s.1": 2}
152+
{"s": [1, 2, {"w": 1}]}
153+
```
154+
155+
`{"s.2.w": 1,"s.0": 1,"s.1": 2}` change to `{"s":[1,2,{"w":1}]}`
156+
157+
### -s, --separator
158+
159+
separator used to combine the keys in the tree
160+
161+
default separator is **.**
162+
163+
### --safe
164+
165+
on safe mode, use escape separator to avoid confilct
166+
167+
expand:
168+
169+
`['aa', 'bb', 'www.xxx.com']` --> `'aa\\.bb\\.www.xxx.com'`
170+
171+
restore:
172+
173+
`'aa\\.bb\\.www.xxx.com'` --> `['aa', 'bb', 'www.xxx.com']`
174+
175+
## Development
176+
177+
Lint and format:
178+
179+
```bash
180+
ruff check .
181+
ruff format .
182+
```
183+
184+
Run tests:
185+
186+
```bash
187+
pytest
188+
```
189+
190+
Type checking:
191+
192+
```bash
193+
mypy jsoncsv
194+
```
195+
196+
Build package:
197+
198+
```bash
199+
python -m build
200+
```
201+
202+
## License
203+
204+
Apache License 2.0

0 commit comments

Comments
 (0)