balm_bulk_insert

1. Description

Balm_bulk_insert is used to clean data for given asondate and load balm aggregators aggregated output data to required 13 BALM tables(inflow/outflow/producttotals) present in ORACLE/MSSQL DB depending on data-type parameter type. Note: The data-type can be DIM(uploads to DimMasterInflowN,DimMasterOutflowN and DimMasterProductTotals), WODIM(uploads to MasterInflowN, MasterOutflowN and tblProductTotals), CONV_WODIM(uploads to MasterInflowConvertedN, MasterOutflowConvertedN and tblProductTotalsConverted) and CONV_DIM(uploads to DimMasterInflowConvertedN,DimMasterOutflowConvertedN and DimMasterProductTotalsConverted).

2. Program location

<installation_path>/balm_bulk_insert

3. Input

3.1 Mandatory Parameters

#	Parameters	Description	Example
1	db_name	Database name	ORACLE/MSSQL
2	connection_string	Balm database connection string	userid\|password\|ip:port/sid(for oracle) and Server\|db\|user\|pssword(for mssql)
3	schema_name	Schema name	dbo/BALM
4	input_path	Path to the data files to be loaded files. Note: It should end with '/'.	test-bed/data/
5	as_on_date	Date for which data has to be deleted from table.	31-03-2023
6	log_file	Path to write logs	test-bed/log.txt
7	diagnostics_log_file	Path to write diagnostics logs	test-bed/diag_log.txt
8	data_type	Type of data to be processed and load.	DIM/WODIM/CONV_WODIM/CONV_DIM

3.2 Non Mandatory Parameters

#	Parameters	Description	Default value	Example
1	log_level	Level of diagnostics written to the log file.	info	error/warn/info/debug/trace/none
2	diagnostics_flag	The flag that decides whether performance diagnostics will be written to the diagnostics log file.	false	true or false
3	terminator	Separator used in between columns.	\|
4	rowterminator	Terminator used in between rows.	0x0a
5	source_list	Comma-separated source name. Note: if no source are provided, it takes all the sources by default.	""	loans, borrowings
6	option	Additional parameters to be used for sqlcmd,sqlldr & sqlplus.The option parameters should be pipe seperated,containing the identification key and its value.	""	m,2\|l,10

4. Output

4.1 Healthcheck report-1 to 13

13 healthcheck report will be generated at the data_file path for each of the 13 tables (inflow/outflow/producttotals).

5. Process configuration in streamdef table:

data_type = WODIM

  .........
    "processName": "Balm Loader",
    "processId": "1",
    "processBinary": "{SH_PROGRAMS}/balm_bulk_insert",
    "processArguments": [
        "--connection-string",
        "{BALM_CONN_STR}",
        "--as-on-date",
        "{AsOnDate}",
        "--schema-name",
        "{BALM_SCHEMA}",
        "--db-name",
        "{DB}",
        "--input-path",
        "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/balm",
        "--log-file",
        "{SH_LOGS}/IND/BALM/{ddmmyyyy}/balm_bulkloader_log.txt",
        "--diagnostics-log-file",
        "{SH_LOGS}/IND/BALM/{ddmmyyyy}/balm_bulkloader_diag_log.txt",
        "--data-type",
        "WODIM"
    ],
    "processDependencies": [],
    "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/balmALL-tblProductTotals-health-check-report.json",
    "processType": "AGGR",
    "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/balm_bulkloader_log.txt",
    "errorLog": ""
  .........

data_type = DIM

  .........
    "processName": "Balm Dimensional Loader",
    "processId": "2",
    "processBinary": "{SH_PROGRAMS}/balm_bulk_insert",
    "processArguments": [
        "--connection-string",
        "{BALM_CONN_STR}",
        "--as-on-date",
        "{AsOnDate}",
        "--schema-name",
        "{BALM_SCHEMA}",
        "--db-name",
        "{DB}",
        "--input-path",
        "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/dim_balm",
        "--log-file",
        "{SH_LOGS}/IND/BALM/{ddmmyyyy}/dim_balm_bulkloader_log.txt",
        "--diagnostics-log-file",
        "{SH_LOGS}/IND/BALM/{ddmmyyyy}/dim_balm_bulkloader_diag_log.txt",
        "--data-type",
        "DIM"
    ],
    "processDependencies": [],
    "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/dim_balmALL-DimMasterProductTotals-health-check-report.json",
    "processType": "AGGR",
    "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/dim_balm_bulkloader_log.txt",
    "errorLog": ""
  .........

data_type = CONV_WODIM

  .........
      "processName": "Balm Consol Loader",
      "processId": "3",
      "processBinary": "{SH_PROGRAMS}/balm_bulk_insert",
      "processArguments": [
          "--connection-string",
          "{BALM_CONN_STR}",
          "--as-on-date",
          "{AsOnDate}",
          "--schema-name",
          "{BALM_SCHEMA}",
          "--db-name",
          "{DB}",
          "--input-path",
          "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_balm",
          "--log-file",
          "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_balm_bulkloader_log.txt",
          "--diagnostics-log-file",
          "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_balm_bulkloader_diag_log.txt",
          "--data-type",
          "CONV_WODIM"
      ],
      "processDependencies": [],
      "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_balmALL-tblProductTotalsConverted-health-check-report.json",
      "processType": "AGGR",
      "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_balm_bulkloader_log.txt",
      "errorLog": ""
  .........

data_type = CONV_DIM

  .........
    "processName": "Balm Dimensional Consol Loader",
    "processId": "4",
    "processBinary": "{SH_PROGRAMS}/balm_bulk_insert",
    "processArguments": [
        "--connection-string",
        "{BALM_CONN_STR}",
        "--as-on-date",
        "{AsOnDate}",
        "--schema-name",
        "{BALM_SCHEMA}",
        "--db-name",
        "{DB}",
        "--input-path",
        "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_dim_balm",
        "--log-file",
        "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_dim_balm_bulkloader_log.txt",
        "--diagnostics-log-file",
        "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_dim_balm_bulkloader_diag_log.txt",
        "--data-type",
        "CONV_DIM"
    ],
    "processDependencies": [],
    "processReport": "{SH_SUMMARYDATA}/IND/BALM/{ddmmyyyy}/conv_dim_balmALL-DimMasterProductTotalsConverted-health-check-report.json",
    "processType": "AGGR",
    "infoLog": "{SH_LOGS}/IND/BALM/{ddmmyyyy}/conv_dim_balm_bulkloader_log.txt",
    "errorLog": ""
  .........

1. Description​

2. Program location​

3. Input​

3.1 Mandatory Parameters​

3.2 Non Mandatory Parameters​

4. Output​

4.1 Healthcheck report-1 to 13​

5. Process configuration in streamdef table:​