2022-06-21: Jamie Lentin Version 7.3-1 * Improved numerical accuracy when calculating weighted mean/stddev 2022-06-07: Jamie Lentin Version 7.3-0 * More helpful error messages if taxonomy (e.g. vessel) items missing * Support DuckDB 0.3.4 (unit tests weren't order agnostic) 2021-11-21: Jamie Lentin Version 7.2-0 * Fixes to support DuckDB 0.3.1 (Bugs in dbWriteTable type casting) * Add mfdb_dplyr_table(), for querying any table * Fix selecting from taxonomy tables via. dplyr 2021-10-09: Jamie Lentin Version 7.1-1 * Support DuckDB 0.2.9 and 0.3.0 (fix tests to be NULLS first/last agnostic) 2021-08-18: Jamie Lentin Version 7.1-0 * mfdb_dplyr_* now supports nested taxonomy columns, e.g. trip_start_port_latitude * DuckDB support. Specify mfdb('db_filename.duckdb') * SQLite support. Specify mfdb('db_filename.sqlite') * Use bulk_copy to speed up large taxonomy imports Schema changes: * Remove taxonomy name restrictions * Change prey.count from INT to DOUBLE 2021-02-23: Jamie Lentin Version 7.0-0 * Add scale_index to mfdb_survey_index_mean & mfdb_survey_index_total * Add 'area_size' as a possible scale_index value * Switch to requiring RPostgres as the default DBI driver * Allow DB params to be supplied with MFDB_* environment variables * Don't require divisions to be made when querying by areacell Schema changes: * Age now stored as NUMERIC * Add trip taxonomy * Add port taxonomy, used by trip for start/end points * Add population taxonomy, for stock groupings * Add vessel_owner taxonomy * Add liver/gonad/stomach weight columns as well as overall weight * Add bait_type/net_type/hook_count/net_count/mesh_size to tow table * Add depth to areacell table * Change schema.count from REAL to DOUBLE (unreleased) Jamie Lentin * Preserve precision when calculating total weight * Inline small mfdb_groups to queries, improving performance * Rewrite Rpath support to use Rpath::create.rpath.params * Rollback transactions after user interrupts * Expose open_ended in intervals for gadget3 * Renumber all taxonomy IDs if there's an overlap with existing ones (not just the overlapped range) 2020-04-10: Jamie Lentin Version 6.2-0 * Documentation example overhaul: Now ~all can be run given a Postgres database * Add single_matches_only option to mfdb_find_species() * Turn down logging level 2020-03-02 Jamie Lentin Version 6.1-1 * Minor bugfixes 2020-02-26 Jamie Lentin Version 6.1-0 * Produce same seeded bootstrap group output on 3.6.0+ * Postgres 11: Don't try checking if existing function is aggregate * Compare REAL to REAL when dividing items up into intervals 2017-10-27 Jamie Lentin Version 6.0-0 * Add mfdb_na_group function to put NAs in the their own group * Remove remaining cs_taxonomy differences, simplify code * mfdb_sample_rawdata now will group into a list, like other sample functions Schema changes: * age now stored as REAL * areacell size now stored as REAL * Allow gear taxonomy to be changed * Allow the species schema to be changed also * Add tow_end_latitude/tow_end_longitude/tow_start/tow_duration * Add gear_mesh_size/gear_mesh_size_min/gear_mesh_size_max (unreleased) Jamie Lentin * Add parameters/suitiability support to surveydistribution 2017-03-14 Jamie Lentin Version 5.2-0 * Add like and not_like to mfdb_unaggregated #46 * Make upper/lower include tail on final group, don't add additional groups to end * Allow open_ended at either end of an mfdb_interval / mfdb_step_interval by specifying c('upper', 'lower') 2015-10-25 Jamie Lentin Version 5.1-0 * destroy_schema = TRUE should drop the whole schema, not just the tables within * Add mfdb_share_with, for sharing case studies on the MFDB schema * Force tables to be created within schema * Document use of MFDB server * Force ssl/user/password when connecting to mfdb database * Let mfdb_import_area import divisions, so mfdb_import_division becomes unnecessary * Add dplyr interface * Allow EwE output to be generated without vessel data * When presented with overlapping IDs in vocabularies remap the ID. * mfdb_area_size can be used to get the entire area size * Add EwE matrix output functions 2016-05-20 Jamie Lentin Version 5.0-0 Schema changes: * Break up case_study's into schemata, ditch case_study_id: This is a fairly major behind the scenes change. Postgres separates tables into schemas. Before MFDB put everything into the default "public" schema. Now MFDB will use a different schema for each case study (i.e. the parameter you give to mfdb()). Whereas before there was a list of case studies that could be used, now you can specify any name to use as a case study. Note that the names are case-insensitive, so "North-Sea", "north_sea", "North-sea" are all considered the same name. This also means you can use pg_dump and pg_restore to backup and restore (or transfer to a different computer) individual case studies. For examples, see: http://mareframe.github.io/mfdb/reference/mfdb-package.html Finally, the mfdb(destroy_schema = TRUE) option now expects a case study / schema name, since it is able to remove just one schema instead of removing everything. To make upgrades seamless, MFDB 5.x will, if the schema is new, check the "public" schema first to see if there is any existing MFDB 4.x data stored under the same case study name. If so it will copy the data into a new schema. In theory once it's done this for any case studies you use you can drop the tables in the public schema to save space, although make sure your data is copied correctly first. If you look at MFDB data via. psql, then you will need to make sure you either set the default schema with "SET search_path TO north_sea", or specify the schema in queries, e.g. "SELECT * FROM north_sea.sample". 2016-06-10 Jamie Lentin Version 4.1-0 * Add mfdb_sample_scaled with ability to scale by tow_length * mfdb_stomach_preymeanweight shoult take the mean of unique stomachs, ignoring prey restrictions * Add mfdb_sample_rawdata to help debugging * Add mfdb_stomach_preyweightratio to get ratio of prey by weight * mfdb_stomach_preymeanweight should take the mean of unique stomachs * mfdb_stomach_preymeanlength should ignore prey samples without count * When importing stomachs, make the default prey count unknown * When importing stomachs, use same weight_total option for prey as regular samples * Add mfdb_concatenate_results to help when you want to concatenate 2 result sets * Allow prey names & coefficients to be specified per-prey * Predator length agg file should be based on the "predator_length" column * Add Saduria Entomon to species list * Add case study IDs for Atlantis models * State which columns are available when cannot find column 2015-11-06 Jamie Lentin Version 4.0-0 * Scale results by tow_length as well as any survey indices * Allow querying by t_group (i.e. country) as well as institute Schema changes: * Remove fleet taxonomy, can group vessels instead * Split vessel taxonomy up into vessel and vessel_type * Add tow taxonomy, with extra columns for tow details * Add t_group to all taxonomies, for one level of nesting * Prey numbers doesn't accept NA #18 2015-10-29 Jamie Lentin Version 3.3-0 * Implement surveydistribution likelihood component * Better gadget_* error messages when column names don't match (or there's no data at all) * Example for fecthing DATRAS Stomach data * Add mfdb_find_species helper to map from other species lists 2015-09-21 Jamie Lentin Version 3.2-0 * Only the length aggfiles should be of the form (min, max) * Add mfdb_survey_index_total * Import functions should allow removal of data * Add institutes for Baltic * Add Deep-water rose shrimp to species list * Add CNR-IAMC to institute list * Remove need for specifying fleetname in data for gadget_fleet_component, it makes no sense * mfdb_step_interval must be integer, it's not going to work otherwise * Add support for fleet files * Fix bootstrapping when multiple group columns are used * Notice if a query function is given a non-list params * Weighted stddev should ignore NULL rows * Weighted mean should ignore NULL rows 2015-04-15 Jamie Lentin Version 3.1-0 * Fix weighted mean function when scaling by an index * Make aggregationlevel in catchbykilos automatic * Allow import of "weight_total", for total biomass 2015-03-16 Jamie Lentin Version 3.0-0 Schema changes: * Allow non-integer counts to be stored, when they have been normalised by haul-length, e.g. * Allow count to be NA, when storing biomass from landings * Make fleet a CS taxonomy, so you can have custom fleets 2015-04-15 Jamie Lentin Version 2.1-0 * Implement weighted stddev aggregate function (ignoring _var columns for now) * Use survey indices to scale count, not replace it * Support grouping of taxonomy items * Allow import/export to tarball as well as directory * Don't write files in one go when bulk exporting * Fix surveyindices likelihood component, should be able to output multiple stocknames * Add gadget_stockfile_initialconditions and gadget_stockfile_recruitment * Add gadget_stockfile_extremes and gadget_stockfile_refweight for manipulating stockfiles * Allow files in subdirectories to be read and then rewritten in the same place * Introduce "bare components" file type, use this to include temperature 2015-03-10 Jamie Lentin Version 2.0-0 * Bulk-load prey data, speeding up stomach import * Should use name to compare new/old items in taxonomies, not id * Give meaningful errors (or just sort it) when lists are handed to gadget_likelihood_function * ANALYZE only our tables, so we don't trigger warnings * Don't match partial names (e.g. age --> maturity_stage) when importing data * Autodiscover min and max for 'all' columns * Only recreate new temporary tables for groups once * Use arbitary 1:100 for 'all' aggregation * Correct spelling of surveyindices * Add mfdb_survey_index_mean, for retrieving acoustic indicies * Query functions should always return a list of data.frames, even if there is no data * Implement catchinkilos likelihood component * Give examples of invalid data if import fails * Add surveyindicies, migrationpenalty likelihood components * Allow specifying an abundance_index when fetching mean length * Add survey_index table and index_type taxonomy, to store acoustic data * Add mfdb_import_cs_taxonomy, to allow changing any of the taxonomies * Remove the distinction between survey data and survey metadata 2015-01-14 Jamie Lentin Version 1.1-1 * Add path to Debian/Ubuntu Postgres socket 2015-01-09 Jamie Lentin Version 1.1-0 * Base aggreations for open intervals and unaggreated data on returned data * Fix iceland example so we only include each subdivision once * Add an example for using bootstrapping * Inspect data.frame columns when making a likelihood component * Allow writing to alternative likelihood files * When grouping by vectors with taxonomies, don't return integer values * Need temperature to 1 dp * More notes on how to install under windows * Rework mfdb()'s ability to guess where DB is * Don't overload as.list for mfdb_aggregates, use an internal generic function instead * Fix sampling for bootstrap groups * Should run pre_query and from_clause for filter_cols too, just in case they are an mfdb_group * Use DB to find out case study ID---might be a remote DB * R CMD check fixes 2014-08-23 Jamie Lentin Version 1.0-0 Initial release.