data-import-scripts

Import data to Elasticsearch

install es module

pip install elasticsearch==8.18.1

run script

python import_elasticsearch.py --dir $DATA_DIR --es https://localhost:9200 --index my_index --user elastic --password $LOCAL_ES_PWD

check index

curl -sk -u elastic:$LOCAL_ES_PWD https://localhost:9200/_cat/indices?v

See mapping (fields schema)

curl -sk -u elastic:$LOCAL_ES_PWD https://localhost:9200/my_index/_mapping?pretty

See index settings (shards, replicas, etc.)

curl -sk -u elastic:$LOCAL_ES_PWD https://localhost:9200/my_index/_settings?pretty

Fetch a few documents

curl -sk -u elastic:$LOCAL_ES_PWD "https://localhost:9200/my_index/_search?pretty&size=5"

Import data to MySQL

create table

mysql> source mysql_table_create.sql

run script

python3 import_mysql.py --dir $DATA_DIR --db_host $DB_HOST --db_user $DB_USER --db_port 3306 --db_name $DB_NAME --db_pass $DB_PASS --db_table=linkedin_profiles

show data

mysql> select * from linkedin_profiles order by id desc limit 5\G

show help

python3 import_mysql.py --help

show progress files

cat $DATA_DIR/.mysql_import_progress.json